Touchscreen devices appear to promise a future of completely portable computing, but it has been a challenge to find an effective alternative to the conventional keyboard. Many portable devices include small keyboards, but it is difficult or impossible to touch-type on these keyboards, so data entry remains slow and cumbersome. Serious data entry still requires propping a desktop or laptop on a surface and touch-typing on a conventional keyboard with both hands. The conventional keyboard is too big to be portable, and the method for rapid data entry on a conventional keyboard is not conducive to the circumstances of portability.
The acuteness of the problem has led to a broad variety of methods for inputting into a touchscreen device. The most prevalent is a virtual software keyboard that emulates a physical keyboard in look and function. Apple iOS and Android OS devices come standard with virtual keyboards. Alternative virtual keyboards are also available that arrange the keyboard keys differently in efforts to improve the rate of user input. For example, a number of virtual keyboards arrange keys to allow for faster input using both thumbs at once. MacKenzie & Zhang (1999) describe a virtual keyboard optimized through an analytic analysis of English. The “Swype” keyboard by Nuance Communications, Inc., also has a keyboard layout, but it does not require that the user lift fingers between selecting characters, thus increasing the rate of character entry. However, input on virtual keyboards remains slow because the entire hand moves when selecting characters and because the user must take some care to position fingers on the touchscreen for each character. In recognition of how error-prone it is to require that users carefully place fingers at specific locations, most virtual keyboards include some form of autocorrection, allowing the user to type at a rate faster than is required for accuracy. To further improve the rate of input, virtual keyboards usually also include a form of autocompletion. Even so, autocorrection and autocompletion themselves introduce errors requiring correction, limiting the rate of input.
Numerous approaches have been explored for reducing error rates, reducing interface size requirements, and increasing the rate of input. Consider handwriting recognition systems. Many such systems been implemented on touchscreens, including a shorthand system that Goldberg & Richardson (1993) developed to minimize the gesturing required. The Graffiti system by Palm, Inc., is a famous example of such a system. These systems generally allow users more latitude for error than a virtual keyboard does, but they are no faster than handwriting. Efficient solutions appear to reside with coded input, such as by selecting keys.
Small portable devices must balance providing enough keys to be useful with providing keys large enough for fingers to accurately target. A common approach to providing character input with few keys is to associate different characters with different sequences of key presses, as done on twelve-key phones. In US 2002/0136371-A1, Bozorgui-Nasbat improves on the traditional twelve-key phone system with a system that only requires two key presses for every character. Frequently occurring characters are input by pressing the same key twice. Less frequently occurring characters are input by pressing a key and an adjacent key. This approach leverages combinatorics to reduce the number of key presses required, but such series entry solutions require that the user carefully target keys one after another. Chorded keyboards address this problem by assigning simultaneous combinations of key presses to different characters. A single pressing of multiple keys inputs a particular character. However, chorded keyboards require the user to learn to move fingers independently, as on a piano, and also to memorize a great variety of correspondences between finger combinations and characters, making chorded keyboards beyond the skill of most.
The ideal solution appears to reside with combinatorics and coded input, provided that gesturing skill and memorization requirements can be minimized. There are other approaches to inputting characters with combinations of coded input. One approach is to initiate a gesture on a first region of the device interface and end the gesture on a second region. The combination of the first and second regions then indicates the character input. In PCT/US2012/064563, Bonner et al. describe a system in which the user touches a region of the touchscreen to select a set of characters. The system then presents those characters in a list and allows the user to drag the finger, without first lifting, to a second region, which corresponds to the character that the user wants to select, whereupon lifting the finger selects the character. In U.S. Pat. No. 5,521,986, Curtin et al. describes a similar approach, where regions are selected in combinations to form characters on a simulated LED by lighting appropriate segments. These multi-region selection approaches have the user carefully positioning both where the finger starts and where the finger ends, thus requiring the user to employ feedback from the system, such as visual or audible feedback, to ensure that accurate selections are made. Perlin (1998) ameliorates the targeting problem a little by establishing all of the regions around the position where the user first places a stylus. The user can then select the first region simply by going in the appropriate direction a sufficient distance. Even so, the user must still carefully target the second region for each character entered, keeping the user's focus on the user interface while inputting characters, sacrificing speed for accuracy.
The systems described so far employ combinations of the same class of input, that class being the selecting of keys or regions. More combinations become available by combining different classes of input. In U.S. Pat. No. 5,341,133, Savoy et al. describe a system of keys that can each be pressed not only downward but also in various directions. The key pressed and the direction in which it is moved together select a character to input. Direction is therefore another possible class of input. Savoy et al. also allows simply pressing a key downward to indicate a character. Twelve keys each supporting four directions of movement and direct pressing can together indicate 12*(4+1) or 60 distinct characters, each requiring only a single user action, which is the pressing of a key a particular direction. In U.S. Pat. No. 5,521,986, Curtin et al. apply this to regions on a touch-sensitive surface. They describe the combination of touching a region and moving a stylus or finger in a particular direction as indicating a particular character, without requiring the gesture to end in a target region. In U.S. Pat. No. 6,104,317, Panagrossi also describes a user indicating a character by touching a region and moving in a direction on a touch-sensitive surface, but further allows the user to simply touch a region to indicate a character, as done with a conventional keyboard, thus providing an analogue of U.S. Pat. No. 5,341,133 for touch-sensitive surfaces. In U.S. Pat. No. 7,694,231 B2, Kocienda et al. also describe selecting characters according to whether the user taps or gestures from a region and according to the direction gestured.
Each independent class of input available to a gesture can be thought of as a parameter of the gesture. The region or key selected is one parameter, and the direction indicated is another. In U.S. Pat. No. 7,057,607 B2, Mayoraz et al. describe several more possible parameters, including how far the finger moves, how fast the finger moves, and whether the finger goes “back” after moving in a direction. Mayoraz et al. have these parameters complementing parameters for region and direction in order to select characters for input. With so many variations available for each region, fewer regions are needed to provide the user with direct access to a full character set. Given the space limitations of portable devices, when there are fewer regions, the regions can be larger, further reducing the probability of user error. However, even with a variety of gesture parameters, all of the combinatorial systems mentioned so far, other than the chorded keyboard, have the user repeatedly moving a hand as a unit to target a finger to a particular key or region. This is analogous to fingers typing by “hunt and peck” on a conventional keyboard, and it is slow. Users can improve their rate of input by developing strategies for where they hover their fingers and for when a particular finger is used, as people do when hunting and pecking, but the user can benefit more when the system itself is specifically designed to accommodate a particular strategy of input.
Ideally, the hand would remain stationary as the fingers gesture, as done when a user touch-types on a conventional keyboard. With the hands stationary, only the fingers need move to gesture, presumably increasing the rate of input. Several patent applications describe this approach. In US20060077179 A1, Chang et al. describe calibrating a virtual keyboard according to the maximal extents of a person's hand on a touchscreen device, producing a virtual keyboard on the device appropriately sized to the hand, with the keys appropriately spaced. In US 20090237361 A1, Mosby et al. describe calibrating the positions of virtual keyboards according to positions registered for fingers and palms, with each hand possibly employing its own virtual keyboard. In PCT/NL2011/050911, Vellinga describes a system that infers the identities of fingers according to their rest locations on a touch-sensitive surface. The system establishes either a user interface widget at each finger or a miniature keyboard surrounding each finger. The character indicated is a function of the identity of the widget or key selected. In US20130113714 A1, Mao describes a similar system that assigns a keypad to each finger. A finger either taps one of the keys assigned to it or simply moves in a direction of a key to select the key. Mao also allows for the system to monitor the directions of movements of the fingers to calculate “median directional angles” for each finger. Mao employs the median direction angles to orient each keypad to align its keys with the direction in which fingers flex and extend, making the keys easier for the fingers to touch.
Each of these hand-calibrated systems associates individual fingers with multiple keys, although Mao allows for a finger to indicate a key, and thus a character, simply by the direction of movement of a finger. Mao includes keyboards that associate some keys each with only one finger but suggests using “more keys per finger in order to cover the whole alphabet.” Each of these systems has each finger of a hand moving independently in any direction. Any of these systems can be conceived in simple form selecting only from two keys along the direction of extension and flexion for a finger, but none is capable of selecting more than two characters per finger under this restriction. Three characters per finger are possible if tapping a key also inputs a character. In order for these systems to provide more than three characters to a finger moving only by extension and flexion on a stationary hand, multiple keys must be stacked along the finger's nearly linear path of extension and flexion. If multiple keys are stacked along a line, the user must accurately place fingers when touching or lifting from the surface, in order to select the proper key, forcing the user to carefully train each finger to target specific locations on the screen. Some users may be able to learn to gesture with such accuracy, but stacking keys probably leaves most users frequently making mistakes. None of these hand-calibrated systems, as described, provides an accurate and efficient means for rapid character input into a touchscreen.
A solution combining several of the above approaches helps to address some of the problems with any individual approach. In particular, extending the Yelling a Vellinga and Mao combination by allowing a finger to input multiple gesture parameters during a gesture gives each finger access to many more characters without requiring the fingers to target specific regions. For example, the direction that a finger moves and the distance that a finger moves can together indicate a character. However, existing solutions that involve gesture parameterization and calibrating for a hand, including Mayoraz et al., have fingers selecting from among multiple keys or regions as part of the process of selecting a character. Additional insight is needed to realize that individual fingers do not need access to multiple keys or regions, that each finger can be assigned a single key or region and yet still provide access to all the characters of an alphabet and more. That is, each Vellinga or Mao keypad need only have a single region, and the user need only input gestures with a variety of gesture parameters; the user need not target individual fingers to multiple keys.
Suppose a solution combines one of these hand-calibrated systems with Mayoraz et al. using only one region per finger. Most of these hand-calibrated solutions place keys to the left and right of each finger, as well as farther from the hand and closer to the hand, and Mayoraz et al. has no concept of assigning keys to individual fingers. The combined Mayoraz et al. solution, given the non-obvious constraint of one region per finger, would supplant keys with directions, requiring each finger to move in any of four directions or more, as with the originally described hand-calibrated systems. Mao includes some fingers that are each associated with only one key, but Mao assigns a letter of the alphabet to each of four directions for each key, further suggesting that the most apparent solution to providing broad coverage of characters at one region per finger requires distributing frequently entered characters over four directions.
Moreover, in a combined hand-calibrated Mayoraz et al. solution, if each finger is restricted to only moving forward and backward in order to keep the user from having to realign the hand between characters, an additional problem arises. Consider the Mayoraz et al. gesture parameters that can be input at high speed: direction, length, and reversing direction. Users can readily be expected to learn to gesture at two gesture lengths, and three gesture lengths may also be possible, though probably with less accuracy. At two gesture lengths, with only two directions and an optional reversal of direction, four fingers can input 4*2*2*2 or 32 characters. At three gesture lengths the number of characters available is 4*3*2*2 or 48 characters. 32 characters are not sufficient to represent 26 letters and 10 digits. 48 characters approaches the number of keys on a keyboard, but it is at a cost of accuracy as users attempt to reliably select among three gesture lengths at high speed. Allowing taps to represent characters increases the count by 4, for four fingers. Mao describes the taps of adjacent fingers also indicating characters, but this only adds an additional 3 characters. Hence, it is not clear that restricting fingers to two directions of movement is feasible in even a hand calibrated Mayoraz et al. style system, further inclining developers to broadly distribute the characters over four or more directions per finger even in this combined system.
Still additional insight is needed. First, it must be understood that a user gestures most rapidly when the hand remains stationary even as the fingers move. Second, an additional gesture parameter is needed that satisfies this first constraint and yet still greatly increases the number of characters available. Because users do not remember combinations of fingers well, and because it takes effort to learn to use fingers independently, chorded keyboards are an unlikely source of inspiration for a solution. Yet chorded keyboards do suggest a solution: the combination of fingers gesturing can further select the character being input. For example, the index finger gesturing in a certain direction for a certain length would indicate one character, while both the index finger and the middle finger gesturing for a certain distance and length would indicate another character. The region that a finger initially touches would identify the finger, and multiple fingers could simultaneously touch multiple regions and simultaneously perform a gesture to input a particular character. The problems associated with chorded keyboards can be avoided by limiting the finger combinations that users must learn. Multi-finger gestures are easy to perform when all of the fingers of the gesture are adjacent, and adjacent multi-finger gestures can be sorted from left to right for a given number of participating fingers, facilitating the association of sets of gestures with sets of characters.
Prior art includes gesturing from combinations of regions to input combinations of characters. In U.S. patent application U.S. Pat. No. 8,359,543 B2, Sengupta describes multiple fingers selecting multiple regions and gesturing simultaneously to input a word. Each region corresponds to a character, and the combination of characters selected together indicates the word, usually selecting a word that includes the characters indicated. There are single character words, such as the words “a” and “I” in English, but U.S. Pat. No. 8,359,543 B2 appears to be directed at simplifying the entry of strings of characters, and it is not clear that the inventor contemplated selecting multiple regions and gesturing to input either single characters or single character words. Even if single character words were within the scope of U.S. Pat. No. 8,359,543 B2, it would not be clear that the invention would be able to input more than just a few characters, because most languages have few single-character words. Moreover, it may not be obvious that an efficient gesture system can actually depend on users gesturing numerous combinations of fingers for frequently entered characters such as letters, that multiple multi-valued gesture parameters can be reliably defined for multi-finger gestures, or that multi-finger gestures can be made more efficient by tailoring the system to the hand.
Finally, there is a fundamental difference between selecting keys and gesturing to input characters that none of these solutions accommodates. When a number of characters are associated with a particular region, the user selects the region to input one of the characters. If selecting a character entails gesturing a distance, as it does when inputting gesture parameters such as those of Mayoraz et al., the finger may move off of its starting region as it gestures. This need not affect the character indicated, but it does inconveniently place the finger for the next gesture, especially if the next gesture is to travel in the opposite direction. The finger must first locate the intended region again before inputting another character. If there is only one region per finger, this unnecessarily slows the user down and reduces user accuracy. Regions should instead be sized to accommodate the lengths of at least the frequently input gestures. That is, the ideal system does not provide each finger with a key (or with multiple keys) but rather with an area for gesturing.